Skip to content

[Fix] [AMD] hicache: Honor UMBP anchor key namespaces - #49882

Open
matthewygf wants to merge 4 commits into
sgl-project:mainfrom
matthewygf:fix/umbp-anchor-namespace-20261111
Open

matthewygf wants to merge 4 commits into
sgl-project:mainfrom
matthewygf:fix/umbp-anchor-namespace-20261111

Conversation

@matthewygf

@matthewygf matthewygf commented Nov 11, 2026 •

Copy link
Copy Markdown

Motivation

A multi-tenant platform can serve customer fine-tunes of the same base model as
tenant-a-support and tenant-b-support, with separate workers sharing a
long-lived UMBP cache across restarts. Common system/tool prefixes can produce
identical token-page hashes and equal-sized KV objects. UMBP omits the configured
model/backend prefix from primary KV keys, allowing a fresh B worker to restore
A's bytes despite their different served-model names.

For page hash h at TP1/PP1, both models previously addressed h_0_k. With the
fix and no extra backend tag, they address tenant-a-support_h_0_k and
tenant-b-support_h_0_k; V follows the same rule. Same-model replicas retain
sharing. Isolation requires distinct resolved prefixes; existing model-name normalization
and backend-tag joining are unchanged.

Modifications

Apply existing config_prefix in _anchor_keys(): +10/−1 production lines.
Reads, writes and existence checks use that helper. Reject complete anchor keys
above 255 UTF-8 bytes before I/O so MoRI's SPDK proxy cannot truncate the
distinguishing rank/component suffix. Rank/component suffixes and pointer order are unchanged. Add six CPU unit tests and update two existing physical-key expectations.
Configured instances deliberately miss legacy unprefixed objects and recompute;
valid empty-prefix keys retain their existing format.

Related: #31920 added
model-aware key isolation to Mooncake Store. This change applies UMBP's existing
prefix to its anchor keys.

Accuracy Tests

Native correctness and latency comparisons used:

  • Source: before b174f4bb4fe0; after 07d36b1bc8f5, mounted read-only.
  • Container: lmsysorg/sglang-rocm:v0.5.23-rocm10-mi35x-20261031,
    verified against the tested digest.
  • Bundled runtime: Python, Torch and native MoRI used unchanged.

CPU/native byte-level cache correctness is measured. Native runs
use real CPU host pools and MoRI standalone DRAM, with one model-only and one
backend-tag-only separation case. Each synthetic page is 32 bytes:
A's K/V are [100..103]/[120..123];
B's are [200..203]/[220..223]. Fresh processes perform seed, probe/write and
independent reads.

Native check Before After
B misses A before B writes; destination unchanged 0/2 2/2
B's put reports success 2/2 2/2
Fresh B recovers its own exact bytes after both writes 0/2; reads A 2/2
Fresh A retains its own exact bytes after both writes 2/2 2/2
Configured reader rejects legacy unprefixed objects 0/2 2/2

Six CPU unit tests cover model/tag isolation, legacy keys, depth-aware writes,
component ordering and key-length rejection. The added boundary tests exercise
186/187-byte prefixes with ASCII and multibyte text, the longer envelope suffix,
and rejection before any client call for a mixed valid/invalid anchor batch.
The two boundary methods fail on the previous PR revision.

  • All 14 existing native UMBP tests pass.
  • All 185 selected cache CPU test files pass across the initial run and reruns:
    170 passed initially; 15 passed after correcting the test environment
    (memory budget and the existing parameterized test dependency). Those 15 also pass on the
    previous PR revision. GPU-only cases in that CPU suite remain skipped;
    GPU serving checks are reported below.
  • Changed executable production-line coverage is 100% (5/5).
  • Native SPDK execution remains NOT RUN.
Run the CPU unit tests

In a CPU-capable SGLang environment, from the candidate repository root:

PYTHONPATH=python python3 test/registered/unit/mem_cache/test_umbp_store_namespace.py

The tests use real CPU KV buffers with a substituted MoRI client. They require
no GPU, model weights or standalone server. The native results above come from
a separate run with the real MoRI backend. To verify the failure before the fix,
run this same test file with the previous revision's python directory on
PYTHONPATH.

Test runtime: Python 3.12.3, Torch 2.11.0+rocm10.0.0 and a read-only
blake3==1.0.11 package mount satisfying SGLang's existing dependency.
This PR adds no dependency. The tests are registered in base-a-test-cpu CI.

GPU accuracy and cache checks (2026-11-12)

Ran on AMD Instinct MI355X**, using Qwen/Qwen3-8B BF16, TP1. Both revisions use Triton
attention, decode graphs, 64-token cache pages, a 4 GB host cache, and a fresh
native MoRI standalone DRAM service with 8 GiB capacity; SSD and hugepages are
disabled. The local runtime image ID is
sha256:e7a2b52f2fcfe9b0275ffe6bee00f423d576fecd428980f6511860e51e9b4ab6,
with Python 3.12.3 and Torch 2.11.0+rocm10.0.0; native packages are unchanged.

Ran the contribution-guide GSM8K sanity check with sgl-eval 0.1.3 on the same
frozen first 200 official test examples: temperature 0, seed 0, 16 threads,
2,048 maximum output tokens, and Qwen thinking disabled.

Check Baseline PR head
GSM8K accuracy 190/200 (95.0%) 191/200 (95.5%)
Request errors 0 0
Truncated/empty answers 0 0

Independent replay verified all 200 IDs, labels, extracted answers and scores
per revision.

Native GPU cache checks restore 960 tokens after flushing L1/L2 and after a
same-namespace worker restart on both revisions. A fresh worker with a different
served-model namespace incorrectly restores 960 tokens before the fix and
correctly misses after it. Three natural-language fixtures (1,153 input tokens
each) return the correct answers and identical greedy token IDs across cold,
device-cache and UMBP-restored states on both revisions: 9/9 responses per revision.

Speed Tests and Profiling

This is a correctness fix with measurable key-processing cost. CPU/native
storage-call latency and GPU serving results are reported below. Timing compares
valid same-namespace warm reads and unique writes. A parent's incorrect
cross-model cache hit is not a valid speed baseline for the candidate's correct
miss/recomputation.

Measured on a shared host in the same LMSYS container:

  • Workload: real MHA pools and MoRI standalone DRAM; 8 KiB/page; batches of 1 and 16.
  • Repeats: four runs per revision, varying which ran first, with a fresh process and service each time.
  • Samples: 20 warmups followed by 100 timed calls per case.
  • Reporting: median of the four run medians and their min–max range, in µs/call; positive Δ is slower.
Operation Pages Before µs (range) After µs (range) Δ
Exists hit 1 28.608 (26.349–28.945) 28.280 (26.059–28.895) -1.14%
Get hit 1 33.522 (33.109–37.714) 37.742 (34.849–39.705) +12.59%
Put unique 1 66.654 (64.404–69.809) 69.864 (65.824–88.194) +4.82%
Key helper 1 0.452 (0.448–0.457) 0.687 (0.679–0.692) +51.89%
Exists hit 16 31.357 (31.084–40.839) 36.859 (34.584–41.944) +17.55%
Get hit 16 84.709 (84.194–86.118) 91.266 (88.599–107.313) +7.74%
Put unique 16 174.060 (168.913–183.002) 182.464 (178.742–214.867) +4.83%
Key helper 16 1.472 (1.445–1.528) 2.941 (2.911–2.997) +99.83%

All 5,600 raw samples/blocks and source/native hashes were independently audited.
Byte/canary checks are outside timing. The helper is measured separately in 50
blocks of 200 calls after ten warmup blocks.

GPU serving benchmark

Using the GPU/model/UMBP setup above, followed the contribution-guide
sglang.benchmark.serving workload: 80 requests, concurrency 16, exactly
256 input / 32 output tokens
, random dataset backed by a pinned ShareGPT
corpus.
Three measured repetitions per cache state use seeds 42, 43 and 44; two full
warmup runs per revision are excluded. Cold runs flush L1/L2 and use unseen
inputs. Restore runs repeat the same inputs after a 15-second write drain and
another L1/L2 flush. There are no additional per-run warmup requests.

Cache state Metric Baseline PR head Change
Cold Output tokens/s 1904.6 1888.2 -0.86%
Cold TTFT, ms 85.35 85.95 +0.71%
Cold Inter-token latency, ms 5.806 5.870 +1.10%
UMBP restore Output tokens/s 1685.6 1701.3 +0.93%
UMBP restore TTFT, ms 123.82 120.53 -2.66%
UMBP restore Inter-token latency, ms 5.748 5.668 -1.38%

Values are medians of three run summaries; latency rows use each run's mean.
960/960 measured requests succeeded, with no errors and exactly 20,480 input /
2,560 output tokens per run. Cold runs report no cached tokens. Each restore
run reports 15,360 storage-cached tokens (75%), zero device/host cached tokens,
and storage_backend=UMBPStore.

Output-throughput ranges are 1892.6–1913.3 versus 1859.7–1896.5 tokens/s cold,
and 1414.0–1741.6 versus 1492.6–1739.2 tokens/s on restore. No clear serving-speed
change is established for this workload. The host was shared, revisions ran
sequentially, CPU/NUMA affinity was not pinned, and the workload is small.
Native restore is slower than recomputation for these short prefixes.

Checklist


CI States

Latest PR Test (Base): ❌ Run #43494411547
Latest PR Test (Extra): ❌ Run #43494411365
Latest PR Test (AMD ROCm 10): ❌ Run #43494411526

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@github-actions github-actions Bot added the hicache Hierarchical Caching for SGLang label Nov 11, 2026
@matthewygf
matthewygf force-pushed the fix/umbp-anchor-namespace-20261111 branch from a6e4e12 to 9a78f33 Compare November 11, 2026 16:17
@matthewygf
matthewygf marked this pull request as ready for review November 11, 2026 16:19

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hicache Hierarchical Caching for SGLang

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant